Consensus Graph Representation Learning for Better Grounded Image Captioning

The contemporary visual captioning models frequently hallucinate objects that are not actually in a scene, due to the misclassification or over-reliance on priors resulting semantic inconsistency between information and target lexical words. most common way is encourage model dynamically link generated object words phrases appropriate regions of image, i.e., grounded image (GIC). However, GIC utilizes an auxiliary task (grounding objects) has solved key issue hallucination, inconsistency. In this paper, we take novel perspective above: exploiting coherency language modalities. Specifically, propose Consensus Rraph Representation Learning framework (CGRL) for incorporates consensus representation into pipeline. learned by aligning graph (e.g., scene graph) consider both nodes edges graph. With aligned consensus, can capture correct linguistic characteristics relevance, then grounding further. We validate effectiveness our model, with significant decline hallucination (-9% CHAIRi) Flickr30k Entities dataset. Besides, CGRL also evaluated several automatic metrics human evaluation, results indicate proposed approach simultaneously improve performance (+2.9 Cider) (+2.3 F1LOC}).